The Evaluation of Question Answering Systems: Lessons Learned from the TREC QA Track
نویسنده
چکیده
The TREC question answering (QA) track was the first large-scale evaluation of open-domain question answering systems. In addition to successfully fostering research on the QA task, the track has also been used to investigate appropriate evaluation methodologies for question answering systems. This paper gives a brief history of the TREC QA track, motivating the decisions made in its implementation and summarizing the results. The lessons learned from the track will be used to evolve new QA evaluations for both the track and the ARDA AQUAINT program.
منابع مشابه
Reflections on TREC QA
The TREC (later TAC) Question Answering track reinvigorated the question answering research community, fostering extensive research on different question types and finding answers in different kinds of corpora. Parallel evaluations extended the research further to include a variety of languages and media types. In recent years, the TAC QA track evolved into the Knowledge Base Population (KBP) t...
متن کاملThe TREC-8 Question Answering Track Report
The TREC-8 Question Answering track was the rst large-scale evaluation of domain-independent question answering systems. This paper summarizes the results of the track by giving a brief overview of the di erent approaches taken to solve the problem. The most accurate systems found a correct response for more than 2/3 of the questions. Relatively simple bag-of-words approaches were adequate for ...
متن کاملWeb Based Pattern Mining and Matching Approach to Question Answering
We describe herein a Web based pattern mining and matching approach to question answering. For each type of questions, a lot of textual patterns can be learned from the Web automatically, using the TREC QA track data as training examples. These textual patterns are assessed by the concepts of support and confidence, which are borrowed from the data mining community. Given a new unseen question,...
متن کاملOverview of the TREC 2003 Question Answering Track
The TREC 2003 question answering track contained two tasks, the passages task and the main task. In the passages task, systems returned a single text snippet in response to factoid questions; the evaluation metric was the number of snippets that contained a correct answer. The main task contained three separate types of questions, factoid questions, list questions, and definition questions. Eac...
متن کاملUniversity of Lethbridge's Participation in TREC 2007 QA Track
Question Answering (QA) is retrieving answers to natural language questions from a collection of documents rather than retrieving relevant documents containing the keywords of the query which is performed by search engines. What a user usually wants is often a precise answer to a question. For example, given the question “Who won the nobel prize in peace in 2006?” what a user really wants is th...
متن کامل